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Computtr-basad butorlng/Qoaohlng eyatems ha^e th@ promf^ of enhanolAg 
tha eduaational yalua of ginalng 'anvlronmentar by guldin#\a studQnt'& 
'dlaoovtry laarnlng, Thia/papar provldas an ln*dapt^h vlt« of i) thfe \ 
phlloaopny toahind auoh aystama» 11) the ^ kinds of diagnostic ^ aaodallng . . t 
atratagiaa , raqulred to Infar a student* a shortaomlngs from obr^rvlng hla - 
bahavlor and 111) the range of ©xpliolt tutorial stpateglis needed for 
diragtlng tha Tutor to sajr tha right thing at the right tlmt* Examples of 
, thasa isauM ara drawn f or^ a oonpufcer-basea GOachlrJg systam for ^a aimpla 
'gaais -» How tha Wast was Won* Our Intefntlon^ln writing this paper lii to 
make axp^iol^ the vaat anbunta of tutorial knowledge riquired . to oonat|uQt 
a ' ooaonlng systam that is robustp friendly and Intalllgant enoUgl to 
survlva in home \or clissroom use.. During the past thra© yaaris, we* ttaV^!e ' 
wltnaased how subtlt tha Qompufcernbasedooaohlng problem really is. Wa 
hopa this paper conveys apmt of these subtiatits ■ — ^ many of which continue 
to resist geheral solution* , , 
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id^as when ^the student appe\|^ to have- none. Thia is slgnlfiGant 
challehge requiring roarty of the 'skills analogous to those of a"^^ ooaeh or 
laboratory inatruGt^, The tutor or coachC 1 ) must be peroeptive "^nough to 
make7 relevant ooraments ^but not bM intrusive a^to deatrqy the: furf- inherent 

in tiie game* Thl& paper preaents one such ooaehing system (named WEST) 

- I ... . 

built abound the gajne "How the West was Won."^^^^he system eKarained as an 
instance jf a g#narift paradigm, called "Issues and Examplea," for building 
such systems* Aspects of the system are exa^ned to dlseover the 

(/) This usage off the term "eoac^ was originated b7 Goldstein^ 1 977 ) / We 
originally conceived of this Rdnd of r tutorial resource as a congenial 
"tutor" but the images *evoked by 'the term "tutor" ^ hava proven\ to be 
inappropriate. In this paper" we ^ shall use '"coach" to emphasize the 
infbrDlai nature of the learning situation ' . = 

' ' -i . ' 



Thf revolution in peneonal Qomputing wlll^ bring With It oxtensiv© use 

of oofflplex garats. Students will play oomputer basid gam^s during muoh of j[ 

t,h©ir free, time* These adtivttiea can prO^de t-rloh , ^,nf^rpaL enVlrdhmenta 

for learning . ^ Oaro^s provide an entioing probiQm-s.ol^lng Ironmant that , a ^ 

'student txplorea at will, free to oreafte hia own IdiaS 9f und^rlylnf 

struotu^re and to 'invent his own strategies for utiU^lng h&ia uh^rstandlng 

of this structure. Properly oonstructed games dan lead to the formation of 

strategies and knowledge str^etures that ha^e general usefulness in other 
- ' ^ ' " , 4. / .. .. ^ ^ * , " • 

.domains aa well; However ^ a mtjor sttimbl'ing block to the effective ^ v- 

educatipnal use of unstructured^ gaming ^r open-ended prbblem-aolvlng x 

environments Is the amount' of tutorial resources that' are often required 

(1) to keep the student from forming^ grossly Incorreot models of the 

underlying structure of tH^ gape/enyironment , (2) to help him- see the 

limits of rtis strategies, and (3) to help him d iscover the causes of ; j 

mlanifested errora. ? * \. ^ 

'1 ' - One of the' prerequisites ' for a productive , inforh'al learning 

environment is that, it be enticing to the student by enabling him to 

control It. ^ The student must have the freedom to make debisionp (incorrect 

as well as correct ones) and observe their results, . While k student's . 

Indorrect deel^ions' sometimes lead to erroneous results that he can 

fmm'e41&tely detect, they often produce symptoms that' are beyond his ability^ 

to recognise , For an informal environment to ^ be fully effective a^s a 

learning activity, it of ten ^. must be . augmented by^tutorial guidance that 

r(ecogni2es and explains weaknesses In the studetTt^s decisions or auggests 



llraltatlonfl ^ of^ th© Qintril paradigm and to oharaoteriie a wide variety of 
tutdflal atr^tegiea* that must Belnoludea to breatf a auoaesafui ooaQhing 



system. * ^ 

CQachlng ^npb^uational Svatems . ^ 

' The pidagoglQal motivation underlying muoh of our GoaQhlng reaearah 
can be Qharaoterl^Qd as "guided dlsoovpry learning*" It aasumes a 
QQ nfit r tt d tivi .gt poflition, in^whioh the student QonstrUGta his new knowledge 
rf?o'^ hilB^existing knowledge* In this theory, the notion of migGonQeption 
or\ "bug" plays a central 'roie. .Ideally, a student's bug will gause an 
erroneous result that he will notice. If the student has enough information 
to determ'ine what ^oaused the error and can correot it,^ then the bug is 
refer'red' tov as 'oons truQtlve > If, howoverj the student does not have 
sufficient information to Ghange his behavior as a reault of the perceived 
error, the bug la termed ppp gona tructi ve\ One of. the moat important 
aspects bf a learning env ironment is the degree to which the mlstakea that 
a ^tuder\^t rtakeg are constructive (See Fischer, Brown, and Burton, 1 970 fo^* 
further d i'soussion , ) . From this point of view, one of the major tasks of a 
coach is to give the student additional information in order to transform 
nonconstructive bugs into eonstructlvo ones. An additional task for the 
Goaoh, in dealing with bugs that do not have easily observable 
manifestations, Is to point out that something can be improved,(2) 

I A subtle requirement 0^ this' theory la that the -coach not interfere 
too much. While the student is making mistakes in the environment he is 
also experiencing the ^dGa of learning frb^i his mista^e^a and discovering 
the means to recover fnom his mistakes. If the coach imjnediately^polnta out 
the student' s-^ errorV'^^ft^re is a real danger that the student will never 
develop7:|he necessary akills TtfF^amining his owh behavior and looking for 
the causes of his own mistakes. 

There are two major but related' problem^s that must be s^olved by a 
computer oo.ach. Th^^y are^ * , j % 



) when to interrupt the student's problem solving activity, and 



XTf Tn a ■ recent paper^ oa The educational implications oT Piaget* a 
psychological theory , Groen has identified similar requirements; "A child 
will learn only if he extends the range of hypotheaes he can generate, and 
raodiTiea or. ellmtnatrfs the transformations that lead to false ones, Ihus, 
it is part of the teachdr's task/tQ ensure that the child, is aware of 
anoinalles and Gounter-examplea that result from his activities," (Groen*' 
1978) 



Z) what to say onom It has been Interrupted* 



In general, flolutions .to these prbblems , requ Ire teGhnlquea}for rie terminlng 
what the student l^nows (procedureB for nbnntruoting « dlafnofltlo model ) as. 
well as explialt tutoring prlnaiples about Interrupt'ing artd td^slng. 
These, in, turn, require s theory of how a etud^nt forma abstraolElonfl t how 
he learnaS-^. and whfin ho In apt to bo mont reeoptlvi? to advloe* 



UnfortuhatQly I few, if any, nx Luting pnyoholog loal theorlofi aro preolse 
enough tq auggeat anything morn than crautlan/ Tho rnqulrfimontB ^that evolve 
from coaching aynlRma nhoulcl prov^in unnful forning funatloqn for futuro 
oognitlve theoHon. In ndflltloni tho cM)nnhing nyntnmn thnmfiel vn.^ ahnijld bn 
good test nnv irnnmentn for (nunh thnorlnn. 

V " 

DlaxnoatiQ Modelltiit 

■ ' . \ 

J)lnan the ntudnnl In prlm/irily nngiignti in a gfimltig or ^ prcib 1 nrn-no 1 v 1 ng 
/ictlvlty, , any- nxpllnit. f\ Vnf^jion \ of n nttjdnnl'n Mt.rongt,hn nnd wonknnnfinf} 
fflu^'it bn tJnofel^up 1 vn or* nubn^fv i f»nt. to bin friHiri nclivity, th 1 fi frmnrifi thnl 
thfi diagnoDtio rM)mponr?nt nrinnot ' unn prontorrKi tontn nr ponn n' lot nf 
dlagnontlc qunatlorin lo thn nt.udont ^ Inntf?;id, t,hn nomputnr^ f'OHfjh munt 
restrlnt Itaelf mnlnly^ to jnrnrring n ntudent'n nhnrtn'nm 1 ngn from whntnvpr 
he donn In the nontoxt' of f)Uiying tJir gfimr' or ^^nlvin^ t.hf* prnblom. Thin 
can be^ a difrinult protdluiti, Jiint brMKiunn a nt.udnnt^ (k)(!/i not unn n (MM'tain 
3 k 1 1 I 1 1 e p 1 n y i n g n g a m o f i o r • n n o t. m n n n h n t h n f 1 n n n no t k n n w . t. h n t, n k i 1 1 * 
For oxnfTjplc, an opponent rnny nrvtM' havo ,nf*ontnfi a nit.unt. Ion thnt, vrqiilvm] 
tiirn to Invokr It* AIt,hcHjgh t.hin point ruM^mri quitc' obviotin, It. ponnn a 
serious clLygnontln problf^mt 'Tho abaoruM: of a rn/iri 1 rf?at<Ml ak I 1 1 f*arr'ioa 
diagnontlo valur? If and only if an '^xpor^t in an Mfju i.yal nnt. nltuat. loti would 
have uaod that nklll. fif^niM^, apart from thr oiit.r^ht nrr^nr^a, \.\]v main 
window a computer- baaed Coach haa to a^atudcnt'o mi aconqept 1 ona ia through 
a "differential" modeling tof^hnlquc^ that nomparf^a what thn nttidr'fit^ la doing 
with what tho expert wouJd<^bo rlolnj^ in hia^plaf!o. Th Iri 'M I f.f n^^M^M> " muat 
provide hypothenea about what thf atudr^nt dona not know or ha a not yot 
rnastered , ^ , , 

« The ^rooeno'Df .cnn53t ruf^t 1 ng a d l f rofMni 1. 1 a 1 mod?'! r^'M^ulr^ia t wo i n-\k:i 
bc|th of whlo^h une a nomputor-ba:uMj Kx[H^rt. jjut, for- (li4'fi'ront p\irp(>r\nn , 

The first teak la evaluating Uu' cpjallty.of the ntuf1f*nt. * fi (Mirmnt /kM. lc)n or 



( 3 ) From here on , tho'ter*m Kxpert wl tl bo untid to r'of»u' t o the nimulaliun 
of 'an expert player. In the nompiJt?M*, * ' ' 



^»oy«" In rtirftlo;iihip to the att of possible altirnatlvf fflovfea that an 
Bxpfert Might havrt fiiAdi 4n tht #3ffldt aaffle dlf^auwfltifidefl , Th« aeodnd task la 
determining the underlylhg flkllla that went , Intrj , thfe feleatlcm and 
oompoflition of) the stu^^pt'e move as well as eaoh, of the **be4tor" moveo of 
the Expert. tn order to wnnompi 1 sh , th^ f Irat -^gk , tftf? Rsfpisrt n^nd dnly 
use the JCiJLUli «r Mn knowledge and rnannnln|5 ntnnt^iftSi Whinh in In th« 
form of bf^t t^rf inov*!n * 'flnw^vpr, for thi? flpcmnd ta^k , It \^f% to nonnjdnr the 
p I enn.i*' ' ^.^^ know 1 nrlgp Invfjlvnd^ in n^l^intlng flnd g^nnrRting thn bnttcf' 
fflov'^jn , ninrn thn nbnnnnn nf nnn f)r f.hnn« plnncn of knnwl r^dg*** might, nKpUiln 
why thn nf. ufinrit, fnlln?) tfj mnkn ^.i ()^t,t.nr mnvp , 

twf? ffM'mn , ■nnr^lfnr'm in ;\n n "k 1 n n n-- box " ^\V\ \ (uil -ttr* mfnlf^l ( f} o 1 fj nt.n In ^ 

pApprt, 1977). Thr^ mndol 1 fi rl^f nrrnd tn' nn /" nt^t iculotn" bncaunf! ^nQh 

prob 1 cm- nr> 1 V I ng fifu-inlun it, mnknn ^^an, f ri pr I d n 1 p 1 ^ 1)*^ PKpWHnnd' in t*M*nin 

thnt m»it.nh (nl nt^m^^ 1 1 of a b hi raO t i n ri ) bhfj^f* uf a human 

prob 1 run-^ ru) 1 vr^r- . ( U ) In f'()ri t r- * -i t. tr) tbn art ifMihit^p Kxpnrt InM-tin "/Tl an k-^ t)oK " 

iLXpprt. , whlc!ti dfita m t [mi t.u rpB HrK||^. pr()fMi n a I ng^a 1 gur 1 t timn ItiaL du nol 

mimin thnru^ unnM by fiumari NM*riKn/ Kor^ ^^xamplf^, Lho qirv^uii' nimiJ'lat,nr 

nndnrlylnM :jOIMnK^l (fifMJwn S Fu.irlon, i a " b 1 n n k - t)n x Kxpn^r t. ^' nnri In 

i/nfMi only t f) ('Jiiw^k t ^H* fori ;i i ri ti/'ri'' y fjf n t iifl *ui t- * n hypnt.hnnnn .irnl arinWfM'. nimr) 

nV bin qunntionn. *lt>n m ni: hur? 1 nma nrn nnvHr^ t'nVf^aifjd thfj 'Jludnnt -ilnoH 

t.hry .irM> t/a i fi I y rwO. t.hf fth^^' fia n i nmf^ t.b'T' nt.ucj "n t> \ rxf'*M't.i>d to .M^qulr^r'. 

Within t;h#i' rrHmf'wn'k of th^* i ^*MfHJnt i t* prnbiomn l anod by t.hn (jumpulfir 

i'oanh, thr*^ glnnn-bnx HxpfM^ nrrm:^ to bo Ih'* mont ursf^fui" airif^u M t. cian b*^ 

ijnf'd t)f>t h rf)r;tb»' i^vnluMt i(?n pr-nfOMM (by Kroirrnt inK opt imal mnv»'M) ;!n<l Cnr 

(1 o r^rn 1 n i n^ th^vM^ilVM ufn! j^f 1 y i n^ thom^ rnovon. :*kill fl r t rm i.nat. i on i/^ 

' . ■ % 

aohinvPd [jy IhnkinK it t h** Kx|\*'!M'n () rMi h 1 ?*m - mo 1 v | rig t rNifM* [ o r* .g "rirq' a I rig a 
givf^n m^)V»^ arw! noting t Ou * t 11 m;; t ha t , it UMrd . Th^ 1 aa:s-^ t)<)X Kxp^O't, la 
alru) tirsofu 1 j ri t.h** »wilijatiori t,aak t>fMj;njan it ran g#"*n t* ^ a t »;> M ?»pa(u^ of 
a i t\fMMiiTi i V -'bntanr*' movfi'rrani :wn'M^ ilrl.i^r^m i rir tJu^ rsink (MMifMMfig of thn* 
1/ I vfOi mi)vV . . Not.*' tiuw';VtM- , that airKM^O.hi^ rva 1. ua t. iuri proof^aa involVr*a 
fl#*t.pr' m I ri i riM }hr rf^npl^tf? , rMn^* t^f ,i i^r ^^r'na t i v b^^baviofvi, il vrqiwju^:^ 
nubat ant i ,i 1 1 y maa^o fManpatat i(Ui^^^ic| raibtia t rw^ar^ than a i m[i 1 y aaaM-\ajng ihf^ 
Mkilln urifi *'r* 1 V I n^^nr^. -^m' .pa f i ' m jjja r- niMV*-,' ;. j _ ..^ 



Slnot the Implettwntation of a bliok-box Expert Is not aoristr*ained by 
humiin-llWe algorlthffls ""It potintlally ban be conalderably = more ef f lalen t 
and, therefore, more useful for evrtluatlon of a student's move, However i 
the akilla It uses to ganirate an optimal move are not analogotis to the 
Student^ It can not be dlf^otly used for the skill detar^minatlon task, 

T*H^ia raises *the prisnlbiHty of Gofflblning an effrclent and robust blaGk^box 
Expert for evaluation with a Lofig efficient glaaa-box Exp^ert for skill 
determination* ' ' 

* , Computa 1 1 ona I af r in lenny In not the only roason for developing the 
Interplay of th«ne two rnrmji of oKpartin^i* The blaok-box' Expert used for 
nvaluatton nend Qnly bn augmnntnrJ with thoae ^nopmpl^te fpieces of an 
arthMjlate KKpr?rt which am nnndnrl to dGtoaj. critical or tutorable f eaty rea 
of thf? ,anrtw*»rn prodiinnfl by thn black-bnx Kxpart". Thd glass-box Expert naed 
not 1)1?. /iblf^ \A) produan thn aomplntn aolu t ion i tsel f , It needs only to 
wnrk hankward n ■ f r^nrn thn nnlutinn to dntnrmlnn ' the ^important" (tutorial) 
fnrMt.urr'n riT tfin., no 1 tit Inn. Tii Ui ronl 1 ?*a t i on opnna up the posaibility of 
conn trua t Ing coanhlng nyntnmn for , domains for whioh we do not Have oomplete 
glnnn-box nx pnrt, 1 nr? . , 

la poaaib /l/ha t <i lot of infprmal learning ocours through the 
fjnmb Inat ion of tank t qx pof^t Inn (in the form or a blaak-boX) with inQomplete 
but. artlculatf^ plmi^ofj of a glaHn-box Exp€?rt, For example', no one has a 
nnmpl*='tf^ a r t i (mj I a to.^lrH^r y on .how to play nxpert ohe^s, AlthOugh^ there are 
nomo fjxcfjiient ch«?ba machinnDj they roly on non>-human strategiea for 
anhln^ijig t Ih* 1 r? k po rM, 1 , t hat \ \ they are black^box exoerts. There are 
a I .no handboo.ks of ohonn . prM nulplerf whlQh reflect pieoes of articulate' 
knowledge* about opcninj^ moves ^ end game tactics, etc. A Qhesa Qoaching 
nyntnin nould t.nkn ndynntfigc} of the blank^box Expert * to Iden ti f y arl t ioal . 
movnn and iine ^ pinoo!3 of Inpnmpleto but articulate knowledge to^ partially 
ex"plaln why the movf* w^vn cnMtinHl and jiow it might have been detected. 



PeopH* alno appear to If^arn natural language through a ^similar interaction, 
A Gofflpietc3, articulatr^ ^ theory or_ English doe a not, exist. People dOj 
howevwr j m an ngr» t^n bpnome fluent In Kngllfl^ by receiving feed 6a Qk from many 
"hlar-k-box ex porta" - oth>n' popple who 3 peak it* To help in the critiquing 
^tank , tliorf' aro Incumploto ar*tLculate piece a .of knowledge, ' such as 
nubJeT't -vf^rb agroomf*n% , That \\\ , in addition to getting black^box feedback' 



thin innuoT^ and'C/UlbON n^ee CI aneny ^ e article TaTThis iasue) systems are 
baaed'- on artioijlate expet^tn, an are many pro^.uotlon rule based experts. 
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of the form "that's not gramrnaticaL , " which could mean almost anything, 
people algd get ^lass-box rules such as "Don't say 'they is,' gay 'they 
are , ' because you must have 'sub J ect-verb agreement . " ( 5 )^ 

The modeling technique discussed in this paper employs a black-box 
Expert in conjuncttpn witb a set of local , glass-box Experts, Briefly, 'the 
bLaQk-box Expert is used to determine the range/ of possible moves the 
st^udent could have made, and the glass-box chunk's of expertise determine 
possible causes for the less than optimal behavior of the student. As 
such, ' we hope this technique might also be useful in providing insights 
into how to transform various black-'box Experts'that currently exist - (such 
as the symbolic integration capabilities of MACSYMA) into interestingj 
educational systems. (6) ' "^^^ 

Tutoring by lasue and Example^-^ A Gen eral- Paradigm 

To' be played well, any game complex enough to be interesting requires 
many differtnt skills. From the^ point of view of a coach, this is an 
important fact ^because it means that when a student does not perform well 
in a. particular situation, it is not neeesaarily clear what skill he Is 
'lacking. The difficulty of determining which skill is being misused is 
. increased by the fact that much of the evidence that the Coach has is 
indirect. That is, the Coach only knows that the student did not make a 
^ better move. From this negative informationj he must determine why not. 
(Contrast this with a situation confronting BUGGY^ in which a bug in a k^d^s 
subtraction procedure will have symptoms.) 



Overview 



The paradigm of "issues and iLxamplefi" was developed to focus a 
coaching system on relevant portions of student behavior and to provide an 
overall .coherence (goal) to the Coaches comments. The important aspects of 
the domain that is, the skl^lis and concepts the student is expected to 

(5) In this case, it might seem that the black-box Expert plays no 
significant role since the pieces of articulate knowledge used to critique 
the sentence could also be used to perform the role of the black-box; 
namely, reject the sentence as being ungrammatical > Howevtjr, the black-box 
Expert also uses tacit knowledge to analyze the sentence in order to 
isolate structural elements (e*g*, nouns, verbs) which are required for the 
articulate mini- theories or principles. We all know the subject-verb 
agreement rule and are very skilled at" recogni zing nouns in sentences, but 
yery_few of us can articulate a prenise definition of a noun* 
(5) The technique might also be useful when there exists a complete 
glass^box Expert that can not do, the problem in "all" ways. For these 
domains it cannot be a^sum^ad that the' student is in fact working the^ same 
way as the expert. ' " 



master are .Iden-tlf led ' as a col ledtion of "Issuea", The Issues determine 
what parts of the student * s ° behavior -are monitored by the Coach, Each 
Issue represents an articulate mini-theory (a piece of a glass-box Expert) 
concerning the structure of the domain. It is characterized by two 
procedures. The first watohea the student's behavior for evidence that the 
student does or does not use its partlGular concept or skill. As suchi it 
is called an Issue Recogn^izer. The Recogniz^ra are used, to construot a 
"model" of the 3tudent*s behavior. The second procedure for an Issue knows 
how to use various parts of the student model to decide if the student is 
"weak"'^ in that Issue. It is called an Issue Evaluator. Thus each Issue 
has associated with it ^ both a Recognizer and an Evaluator as procedural 
spec ial ists . 

At any point in the gamcj the hypotheses concerning the weaknesses of 
the student can bo determined by r^unning'all of the Issue Eva 1 ua tor s . on the 
model. When the student makes a "poor" move, his weaknesses are compared 
with the Issues ^ neoGs^jriry to mnke better moves in order to try to account 
for why he did not mBke a bettar rnqve. That is, the Coach looks for an 
Issue in which the student in lacking and which Is required for the ^ 
Expert's better moves. Once an Issue has been determined, the Coach can 
presnnt an nxplanation of that Issue together with a better move that 
illustrates the Issue, In this way, the student can see the usefulness of 
the Issue at a time when he will be most rHH^eptive to the Idea presented 
Immediately after hu has attempted a problem whose solution requires^ the 
I asue . 

F;^une 1 Is a diagram of th^^ mod e I ing/ 1 u to r ia I process underlying the 
Issues and Examplnn paradigm. Figure la presents the process of 
^onst r^uct ing a model of the student *3 behavior. The model is a summary of 
the, student's perfor^mance wh^le solving a serLo's of problems (in thi'^' case, 
moves in a game), Kacti tLme the student makes a move, the important^ 
aspects of his bejiavior (the Jssues) are abstracted by the R^eognizers. 
Thi£h abstracting is also done over the tuMiavior of a compute r- btfsed Expert 
in the aamcj OnvLr^onment by the uam e HueognLzers. The two abstract ions a^re 
eompnrnd to provide a d L f f c nuJlt Lh I o d u 1 of the student's behavior, which 
indicaten t.hose .Issues on whioh ttu^ student is weak. We reiterNite that 
Without the Expert It is not [lOiuiLble to determine whether^ the student Js 
weak in nome skill, or whether^ tlie skill [ia>.i not been used because the need 
for it has ,ar*lsen Infrequently in the ntudent. *s experience. 



FIGURE 1 
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, FIGURE lb 



Figu^^^^ lb presents ^ the top leve^ of the Coaching process* When the 
student /fnakes a less than oplimal move (as determined by comparing his move 
with jrfiat of the Expert), the Coach uses the ' Evaluation component of each 
Issyfe to create a list of Issues on which the student is weak* From the 
Ej^pert's list of better moves, the Coach^ invokes the Issue Recognizers to 
/dete^rmine which issues are illustrated by -better moves. . From these two 
lists (the "weak" Issues and . the "better move" Issues), the Coach selects 
an ^-ilssue and a good move that H 11 ustrates it, (i,e,, creates an example of 
it) and decddes on the basis of other tutoring principles whether or not to 
interrupt , (7 ) If the ^Coach decides to interrupt, the selected Issue and 
Example are then passed to the explanation generajtors , i which produce the 
feedback to the student, 

1 rm GAMING ^SITUATION | 

"How the West was Won" (WEST) is'^ a computer board game that was 
originally designed at Rroject PLAT0(8) to give students drill and practice 
in arithmetic* The board (see Figure 2) is 70 spaces long* In a turn, 
each player receives three numbers (from spinners), which must be used in 
an arithmetic expression (using the operations addition, subtraction, 
mul tipl lea tioh , and division)* The value of the expression is the number 
of spaces the student is moved along the board, The^'object of the game^ is 
to be the first player to land^exactly on 70. To make the student's task 
more complicated than just making the biggest number, there are several 
kinds of speGial moves. Towns occur every ten spaces. If you land on one, 
you advance to the next /one. There are also shortcuts. If you land on one 
of these, yam advance t/^ the other end of the shortcut, (9) And if you land 
on the space your opponent is ocGupying, he is bumped back two towns, 
unless he is on a towp, " The spinner values in WEST are kept small, so that 
special moves will often be better (get onf further ahead) than making .the 
biggest number*(10) 

( 7 ) If there are no f s s ues Tn common between tfie two 1 rata , thM reason for 
the student's problem lies outside of the collection of Issues, and the 
Coach says nothing*' 

(8) The PLATO game was designed by Bonnie Anderson in Dr* Robert Davis's 
Elementary Mathematic Project (Dugdale and Kibbeyj 1977), 

r9) In Figure 2, Spaces 5, 25^ and are the beginning of shortcuts. 

(10) The rules assumed in this paper are the ones used on the PLATO system 
of 1975* Our coach system, WEST, allows the student^to change many of 

the rules. For example, the board length, the distance between towns, the 
location and number of shortcuts, and the set of* legal arithmetio 
operations can all be changed and the coaching system will continue to 
work. In addition, the number of spinners can be changed, but we have not 
built an Expert for such* Changing the rules gives students the 



A 



•Figure 2 shows a board situation that, illustrates sdfle of. the 
complexiLties of tutoring, even in this simple game. The student ^is at 38 ^ 
his oppc^nent is at 39,(11) aad with his spinner| (2, 1; 2)^ ' the' student 
makes pe expression 2+1x2, resulting in a move of ^. Consider the 
alternative moves the student could have made* he .oould have moved '1 and 
bopped his opponent; he could have moved 2 and landed on 1 town; He could 
ive moved 6 and taken a shortcut* What possible reasons 'may under^^e this 
su.boptimal move? ' / . . 



Stagecoach's turn 

The^Numbers aire: 
Your Wove: 2*1^2 
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Opportunity to see the relationship between the rules and the "/feeling" of 
the game^ . / 

(11) WEST is typically used by one student playing against the/ computer's 
Expert, It is also possible for two students to plaV against/ each other, 
in which c^se differential models are constructed for each atuc|fent, thereby 
enabling coaching for both players. 
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The Issue s in WEST ' ■ ^, . ■ ' • 

/ ' ' " ' / . ' ' 

In the Issues and Examples paradigm , ^ the ffg sues embody the Important 

concepts underlying a student's behavior and define the space of concepts 

the. Coaoh can address, IrfiWESTi thecA* are three levels of Issues that a 

i 

/'Coach can foQUS^on* ^At the lowest level are the basic matheraatloaL skills 
thkt the student is 'practr^ing. Jri the ourre^t system these Inalude the 
use' . of -PARENTHESES, ,th^^ use of various 'arithmetic operations such as 
.SUBTRACTION and .pIVISIO^^' and the form of (the student's move as an 
expression (PATTERN). _ ■ ^ " ' 

The second level ^ncerns the skills needed to play WEST. The Issues 
at this level are: the'special moves of BUMP, T^WN, and SHORTCUT; th# 
directiBn of a > move (for example ^ "both FORWARD and BACKWARD are legal); 
and the development of a STRATEGY for choosing a movej such as maximizing 
the distance you are ahead of^your opponent. - 

At the third level are the general skills of game=playing , -One such 

i 

general skill is the strategy 'Of watching your opponent in order to , learn 
from his moves. Another is the effect that different rules of the game have 
on determining the best at'rategy * ( 1 2 ) 

Each of these Issues is represented^ in two parts: a Recognizer and an- 
^ Eval'uator* The Recognizers are data-driven from the local context of the 
student's and the Expert's moves. The Evali^ators are goal directed (what 
are the student's weaknesses?). The Issue Redognizers of WEST are fairly 
s traigh t f orward J but are^ nevertheless ^ more complex than simple pattern 
matchers. For example, the Recognizer for the PARENTHESIS Issue musf; 
determine not only whether or not parent+iesea are present in the student's 
move (a lexical check of the expression underlying his move) but also 
whether they were necessary (which requires parsing the expression) o if 
they were necessary in the optimal move (which requires parsiiu^., ohe 
expert^s behavior) . 

For the situation shown in Figure 2, the following Issues are involved 
in better moves: Moving. 1, entails knowing about the BUMP rule and using 
SUBTRACTION or DIVISION .(^1 3 ) ^ Moving 2 entails DIVISION, knowing about 
TOWNS, and knowing that the order of numbers in the expression does not ' 



(12) At present the copching system does not address these directly, 

(13) The\ student could, of Gourse. move 1 without being aware that it will 
lead to\ a bump. One ramification of inadvertent raovrs is that the Model 
will oonnain some "noise" » Noise will be disoussed in the section on 
Modeling Methodology* 
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have ' to be the same as the 'spinners, 
knowing' about SH<^RTCUTS. 



The Model In WEST 



MQvlng"§==^-€ntalls PARENTHESES a/nd 



Figure 3 shows some of the fields of a student model 'oreated by the 
differential iodeler* The fields it shows include patterns of moves used by 
the student, speQial move^, parenthesis usage, and strategy censidtrations . 
The columns headed by "M1^S^» or "MISSED" are places where \he Ex,p§rt would 
have used the skill but th^Xstudent did not* They are indignations ^of 
potential weaknesses* The SDHudent shown in Figure 3 appears tp be weak in 
the Issues PARENTHESIS and BUHP. 
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Tutorial Considerations 
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Even= when relevant Issues and Examples have been identified , it may be 
Inappropriate to tutor-. This is , determined-^-\by invoking various tutoring 

' stra'tegies . Figure 3 shows traGes of the Coach's tutorial decisions duri^ng 
t^ interaotion shown, in 'Figure 2. One example is th© 'decision about which 
of the competing Issues to -choose* If Chere . are two Issues, both 
applicable, to a certain si tuation , ^which should be picked? .This is where a 
'!syllabus" (Goldst^inj 1 977 ) might be useful to provide relative orderings 
of importance or prerequisite 1 inks over the space of Issues. However, -the 
Issues in WEST are sufficiently independent that there is little need to 

.Gonsider the prerequisite structure/ Instead, additional ^tutoring 
principles must be Invoked to decide* which one of the set of applicable 
Issues should be used. 

We have experimeTited wi\h two alternatdve principles for guiding this 
dedis,ion thus far. The first is the Fdcus strategy, which ensures that if 
everything else is- equal, the Issue is chosen that was most recently 
d iscussed ;''fthat is, have the Coach haEpnier away on a * part Ocular Issue until 
it is mastel^ed. The alternative principle is ttie Breadth strategy , whichv^. 
ensures that if everything else is equals an Issue is selected that has not 
recently been discussed^ This strate^gy minimizes the chance that a student 
gets bored 'by hearing to.o much about one Is3ue, A simple agenda mechanism 
enables experimentation |; with ^ a, range of =raixed strategies lying between 
either a "pure Bread6#4aE Focus s trategy . ( 1 i4 ) The default' is the Breadth 
strategy, because it prevents one , of two interdependent ' Issues from 
blocl^ing the other. Strategies for manipulating the agenda mechanism 
prov^ide only one source of guidance for the tutor. Additional tutoring 
principles will be examined in the next main section. 

5 nation . * / 

Once . the= dec ision has been made to .tutor on a particular Issue with a 
particular Exg^mple, the Coach still has to decide how to express the Issue 
to the student. This is the explanation problem. It is in general very 

(14) The agenda mechanism is implemented as a priority ' list , ^ along with 
procedures for .^reordering it. When two Issues are possible^ the one that 
occurs finst on the list is chosen, The "focus" strategy moves a' selected 
Issue to ^ the front of the list, making it more likely to be chosen again, 
and the "breadth" strategy moves the selected issue further down the list. 
Since this list can be partitioned into sublists, it is straightforward to 
have one strategy manipulate the sublists and another to manipulate the 
elements within a sublist. 



^^^ifficult. In addition to saying, the things the student does not knpWp 
^^^nversational ^^ostQlates" ^ dictate that things^ the student knows already 
, I shoVll|^.n^ said, (See Clancey*s "ariicle for more of' a discussipn on 
this "'-point . ) ^ In designing '^WESTj we have conoentrated on the student 
mQdiling taslc and the t^sk of dttermirting when -to break inj and *have 
prbgressed very little d^n the explanation problem. Currentl^p the 
explanations are stored in a procedure attaohed to each Issue, called a 
Speaker. Each Speaker is responslBle for presenting . ap few line^ of text 
explaining its Issue. At present, the Speakers work by randomly sedeoting 
preatored oomments* Several iraproveraents should be made to the Speakers, 
For exahple, the explanation should be able to ^handle multiple Issues* .It 
may be very difficult to distinguish. betweetK^twO lasuesj and having a 
Speaker that can assimilate both into one succinct comment ^conveniently 

' side ste'ps the need to differentiate between them* For example, the Issues 
of SUBTRACTION and moving BACKWARDS often occur together and it is 
tfverefore difficult to separate the two* 

Uses of Issues ' = 

While the Issuea were originally conceived of for guiding the 
' critiquing component of the &ystemj they have ^jproven to have ot^^^er 
tutorial usesl.in our system. One example is when ^the student asks for^^elp 
while considering what rapve to make. ^If the best move involves an Issue^-.on 
which the student is Leak the "hint" cati stress that Issue. Oui^^ 
motivation here is that the Issue may be the critical piece of information 
for the student to see how he could make the good move, and hence the hint 
should put ^raphasis on it, 

Issues "i are also useful in determining when to give the student 
po sitive en^oura^em_en_t i thus keiping him from viewing' the coach as being 
only critiG'-al. Our current Encouragement strategy directs the Coach to 
congratulate the student on his good move whenever it is the optimal /nove 
that demonstrates an Issue on which the student is weak* However, as we 
explain next^ no one strategy determines what the CoaGh will\, do beo^se 
different strategies^ may set up competing goals- ^ 

, 2. PEDAGOGICAL STRATEGIES ^ ^ • ' 

There are many principles that spell the^ difference between success 
and disaster in a comput er-based gaming-plus-coaGhing environment for 



■ ■ ■ ' ' ■ ■ ■■ / 

informal learning. Over^ the last few years,, we ha|/e had a^chance to 
expeTiraent ^with WEST, and^modify it in response tp^ various subtle and 
not-so-subtle difficulliis that we have encountered. In thia aection we 
will discuss some of th^principles that we found important to embed in ^our. 
system and identify thoseiwhich hav^ general *ip.^plioahility to, informal 
learning situations^ . For the purposes of our pisouasion, we will 
distinguish two types of prinoiplea those for strbcturing the gaming 
^nvironmeftt itself , and those for guiding the Coaoh within the environment, 
' Although much of what we have . discovered concerns explicit* lea/'ning 
env i romnenjts , we believe that many of these principles are also of 
importance in designing other -^friendly" man-machine systems where the feel 
or ambiance of the total fnvironment (including peripheral assistance or 
b.utoring)iscrucipl. 



Philosoptf 

Before discusaing these principles^ let taa briefly summarize the 
philosophical underpinnings of a coaching environment. In these 
environments it is ^"best for the student to discover for himself as much of 
the atructure of 'a situation' as possible * (1 5 ) 'Every time "the Coach .tells 
the^r^^ud^^^ it is robbing him of the opportunity to diacover it 

for^^ftself. Many human tutors interrupt far too often ? generally because 
of a lack of titne or patience, and they may be preventing the development 
In their students of important cognitive skills the cognitive skills 

that aliow students to detect and use of their own errors. 

However I there are times when interference With the student's 
discovery; process is called Tot-. In gaming situations, an untutored 
(unwatche^) student may fixate on a subset of the available moves $nd hence 
miss thej potential riohn.ess of the game. In WEST, for example, a student 
may adopt the strategy of adding the first two spinners and multiplying the 
result by the third spinnerj (A+fe)*C. Since the third spinner tends to be 
largest, this strategy is close to the strategy of multiplying the largest 
number by the sum of the other^ two numbers (which ^rodUQea the ' largest 
possible result)* A student can , remain at this plateau Indefinitely 
without perceiving the failings of his strategy. But no.tioe how tnuch of 
the atructure of the game ^ is being missed. The student is unaware of 

( 1 5 ) This is hot to say that" structure^d material (~e ~] ^7'^^^Ci%^^boo^^~^ s^mjTtf 
not have a j^role in formal education or that gu_^4pd^. diacovery, learning i.g 
the only way to learn! 
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special moves, such as bumpSj and therefore of such questions aSj "Is it 
better to send my opponent back 1^ or get 9 ahead of him?" S^nce his 
strategy requires no search ^to determine a move, the student misses the 
whole notion of strategy as a raethod for deciding between alternative 
moves. From the point of view of practicing arithmetic, he is perforniing 
one calculation per move instead of the dozens of mental calculations he 
would have to perform to answer questions such as, "What numbers can 1 form 
with 'these spinn^fers?" or "Cart, I^ _^ake a 15 with 9* 10, and 6?** By 
i'piter jepting comments and .suggesting better moves to the students, a Coach 
can greatly expand t^ie student's involveraent in the epvironmentj 

The top-level goal driving the Coach is to ensure that its comments 
are. both\ relevant and memorable , The Issues and Exainples tutoring strategy 
provides a fra^jj^ework for maeting these two constraints* The Issues are 
used in. the diagnosti^c process to identify at any particular moment what is 
relevant. The Examples p/iovide concrete instances of "these aU'stract 
concepts* Providing both the desoriptlon of the generic Issue (a concept) 
ds well as a cjoncrete example of it increases the chance that the student 
will integrate this^-piece of tutorial commentary into his knowledge* 

The. Issue that is raised must be -one in which the student is, in fact, 
having a problani, lest-=^h€j advice be ignored or meet with hostility. . 

Erinciple 1: Before giving advice, be sure the Issue used is one 
^ 1 n w h Ic h t h e s t u d e n t i s w e a k , ' . 

The primary ramif 1 cat io^n of this principle is in how the Evaluators ' use the 
student model. As will' be discussed in the next section, there is "noise"" 
inherent in the^ model* The Evaluators fc^r each Issue must allow for this 
and be "conservative". Another ramification of this principle is that the 
ays t era should be cautious on tutoring an Issue that the student has 
recently been advised on, 

^ V e n if the diagnostic process can g u a r* a n t e e the weakness of an Is s^u e 
at a given m oin e n t , the absence of a ^Q^d_ Example of th^t lssue. should 
prevent the Coach from br^eaking in. Thus one of the ' tutoring" principles 
for enhancing a student's likeliness to remember what is said must 

determine what a ''good example" is: 

f 

Principle 2; When illustratLng an Issue, only use an ICxample (an 
alternative move) in which the result or outcome of that move is 
dramatically superior to the move made by the student* 



Another baa^o principle that inQreasea the Qhange of , remembering the 
oritlclsm of the Coach la to have the student' oplGOdically encode the 
example. ^ " . '\ 

Prin\ciple 3: After giving the student advice, permit him to 
/V inco)rporate the Issue immediately by allowing him to retake his 
turn. 

This principle not only provides hirri the opportunity to observe results 
of making a new move based on this Issue but is also apt to deTorease his 
antagonism to the advice. 

The final principle of this category presupposes that the student is a 
bit competitive and that he is leap receptive to advice when he is about to 
lose (even if he incorporated the pdvice when_ retaking; his turn). 

Principle 4: If a student is" about to lose, only Interrupt and 
tutor nira with moves that will keep, him from losing/ ^ 

interest ^ ' ^ 



In' an informal learning situatton, the student's interest stems 
primarily from the situation itself, A student plays a game ^because he 
enjoys It* He n c e , on o of t h e mo a t 1 mp or t an t Ron 3 1 r a i n t s of t he Co ac h ^ is 
not to destroy the student'n Inherent interest in the game by butting in 
toc^often. It woulci be 'much easier to^ implement a Coach that broke in 
whenever the student mode a suboptimal move and told the atudnnt t.he better 
move. But, facod with such a tutoring strategy, the student woiild quickly 
lose all interest in playing the game especially if he were a poor 

player who could profit frHim Judicious . advice. Below nvi> some of the 
principles incorpor^ated into WK8T to prevent It from be? I ng oppressive, The 
first two principles are t h e [no a t o b v 1 o u s : . 

Prlholple S: Do not tajtor' on twu cunseoutlve moves, no matter 
.what . ' ^ 

P r 1 n 0 1 p 1 e 6 : Do not t u tor b e f o r e t he s t u ci vn t h a s a o hn n c e t o 
discovor the game for himself. 

When a new student first s 1 1 s d t o pi a y the game o who n a s t ud n t . wh o 
has not played, in a while re tujms to the game, he will take some time to • 
f a m 1 1 i a r 1 7. e himself with its m e* e h a n 1 e s * He will b o using e og n i 1. 1 v e 
resources to figure out, for^ exai^ple, how to t/pe in nn oxpresslnn* It is 
unreasonable to ex pr c t him to p e r f o vm a t h I s b e s t when i t c cm e s t o a e t u a 1 1 y 
ohoosing a move before he feels fairly comfortable with the mtuihanles of = 
the game . ^ . . 
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'Prlnnlpln 7: Do not provido only cr^Ltictom whun thcf TuLor brenka 
inl If the student makea an u^uepl ipnai^^movQ ^ Iduntify wtiy it is 
J good and uorigra tul ntn him. ^ 

* , . ' ■ 

In WEST . this is done wht*nt*vor a FAIR p-Iayor'' makoD an optimal move or 
whenever a playor makr^n yin optirml miwf' t.hat unen an Innuo in|whlch he is 
weak. Not?-* t var'ioun w^v\s !> the Kxp^^ra Junt lo carry^^^ut thiy one 
p r i n a L p 1 e . 

Thin next princlpU^ nan appearM;l hc'fnpe in a nlightly difrurt^nt J'orm, . 

Principle 8: \ kVWv' giving- advh**- in thr i^tudent., offer him a 
nhance to rntak^^ hin t.ni^n, -^ut. do not fnrcn him to. ■ , . < 

ir thn student oan uaf^ th^^'utor'a adviuo ti^ LmprovV} hiu ponitlon in the 
g^me, he may bP morr^ a t t. i v r> , hut. ho nh.nild br given, a c ha nco to refuse 
to retake niy tiUMi. ain^;?' he may onnniij or a retake to be "a nubtle form of 

oheat ing . 1 1 h ) . . ^ 

^_ • 

Th" n«'xV t.wo pV i n'^ (> i on ^w*m^o donigned to inoroant^ the chances of 
le'^rning from t.-ho ganiiriK v i run on t IndopondtMit of tho Coaoh'a nommentn on 
tho prM^grHM^n of tdu^ ^atn ■ , 

f*rMnoiplt' n Alwayn [iiv^^ t:nfTiputo[^ Kx|^M't- |)L^"Ly an uptimal game. 

The ataivtont nhould l)o iblr t oh:n^r^vo and loaon from t ho heat- po-uu'ble play 
of hin v^pp^'nmt Mvpiouly t tO' ^-nm [mj t. o rO . Ono oK t [lo beat, inotaakllla that 
a ntaid^-nt. o,in loarai fraon WKSl (of any gaino) in t.o watoh wiiat. your oj^ponent 
ia do Ing,^ oapf^Mally if vnu aro 1, oMin?^.> To rnaKimlzo tho (Mkiuoo cU' tdio 
student HOf'Lng t hf^ valuo ot thin hfMjrMatle, he ^^hould^ a Iways have 'a- nhanruv 
t.o obnrrvo okP^mM^ [^liV. Ai:-s, if ^ bo ntudent. raoilizi^n t^iat t lio ooinput er in 
„fue plavui^ tho b^MU i^'OiMibh' ^^^!no , b'^ may foi*l that ho in tnMng i^layod 
downito \[id Ml s|/(]u oti t ] V heir iinaaa^nl in playing, 

Pr'inoi.[>h' 10: Ir tho ntiidont ankn for^ help, [Mawldo a^'voiMl loveln 
0 f h i nt n . ^ % 

m WiCVr thoi'o ata^ ('our' IrVf^n ot^ bolp. Fli o fifMit raniin'nt lor^ help oauaon 
tho Coat^b.^'t i5 lot)k at. t h*^ ntudmi i[U)ilol t\)r hin tMn'ront, woa k n o^i no n if a 
w«Mki\onn la found in a nti i I I taiat la ian]UiiaMl fen* ail optimal movo at thin 
point in tho gainr , t [u^ ntudont In told to ocMinliio!^ tdiat. Inauo. Koi' 

TT^r Tf '^Wl^HT^^nr b^^ uaod ^fn t eo " mtnTo ""whiuv two ~ '?vtaurenfTi^ ^ro splaying 
agalnnt eaeli otdUM', iho at)ility^ to relakt*^ turiin aftor ady i oo in turuied off, 
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example, ifV the atudent is weak on the PARENTHESES issue and the optimal 

movr for this turfi requires parentheses, the student will - be told "Why 

don't you try to use parenthesep to change the order in whiah operations 

are donu." ^'he second request for help J on the same move provides the 

student with the set of possible outcomes. For -the third request^ the Coach 

will serect jthe outcome that it considers be&t. The fourth reques-t Gauses 

the Coach to give the student an arithraetic expression that, brings about 

the best outcomo. Thus, the four gucces,sive levels of hints are based on 

t he f 0 1 1 0 w 1 n g ^ r u 1 e 3 : 

HintM: isolate a weakness and direc tly address that weakness. 

Hint q (what): Delineate what the space of possible moves is at 
this point in the game, ^ ^ 

Hint 3 (why): , Select the optimal move and tell him why it's 
-optimal. 

Hint n (how): Describe how to make that optimal move* 



|nvi.r:dn?yntal ^Conjiderat ions . ' ' ^ 

'■^frw' /wfilLe moat of the interest in a gaming environment is derived from the 
'itself', many things can bo done to the environmen,t to make It more 
in terent iii-g : . Graphics ia a prime examplG. PI ay Ing ^against the computer is 
- anDtJa^r*.^ (Many CA I games have survived solely on the basis of these two 
cbnaidera t Ions . ) In thie section we discuss some more subtle 
cons Idorat ionn that WEST employs. The next prinQiple attempts to keep the 
n t u d 0 n f r o m g e t ting ( 1 i s c a u r a g e ci . 
' ^■ 

Princbple 11: If the student is losing consistently, adjust the 

1 0 V 0 1 o f p I a y . ' ' ~ ' 

Noticie t.hat thi?, priiujiple conflicts with an earlier principle of always 
having ''.the cDmputet^ P^ay an optimal game so that the student will have a 
mcuicl of expe!M play. Koi- games in which t^here are several levels of 
strueture to the play, such as chess , it may be better for the student to 
have a role model (lionro opponent )_^,Hhi-oh is only slightly above his level. 
Thin will tend to keep the^gamen close while still providing examples o/ 
better movea, Ou( noj/ution of this conflict is to give the computer bad 
apiniUM^^^ when It YJ^ ahead by a amount, that yaries with the quality of tht 
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Vv ino i p 1 e 1 ?. : I f t lu^ atuden t nfiakes a potentially g are less error . 
hv forgiving. But. provide Gxpliolt commentary in oaae it was not 
Junt ^nareleas , , ^ 



The flyst«ffl shculd bi friandl y about a atudejit'a error that . may v b& Trorn 
ttlsintirfratliig. tht rulei of th« game cr from raiatypinig a move, On auoh 
^ppcrp, the systsrf should not anly allow tjbrrstudent to'-oorreot his ralataks 
but j If a gtn ir#l rure-of ^th« gamfe bis /been violated. It should draw 
.att^ntlpti' to tn© ruie^ and provldi sfeolflc inatanoes of It that are legal. 
■ ro^ iiiBiflep tha WEsr system has 0D«ipi.l©d. into it diagnoatio routines for 
'mmy tsrflail mr^Ofm that a atu^ent la apt to make <suoh as preGedence 
errcr^ Ir ari thrtitlo and glv Ing aa the yalije or his Gxpresalon the end 

tpi^osifcton of tha raovi>, ' ^ ^ 

^IttioagK the twelve prlnciplea Uated here are oompiled Into our 
^ystsw^ It- is our hopi, that at gome fuiture time thes€ prlnGiples oan be 
"dl^ccfciy interpreted frpm a declarative representatiori or them. Suoh a 
reDri^entatibn eould preside a met a^eny Ironment in v/hioM student teachers 
y aou Id roodify and oxtond th© rul^s and witness the effeots.on studenta. *In 
WEST, a small adva'nce aaong this dirae'nalon has been, mide by enabling ttie 
Coach tc artj^late mil ^t he pros and cons of what it shouLd do nmKt . Of 
opurs©, the Coaah* 3 aogitatiori 1^ not p^pt of what a player sees as he is 
pUjit^g the 'gmm tout instoad is displayed on segond "soreen" (see ^Figure 
3)\ .Thta trace or tha Coach's behavior provides a graphic Illustration of 
hjSjiF- raanaf of -the above prlnotples interact to prcduce some very subtle 
tutcrl-al behavior* 

3. ANALY SIS OF MODEL INC METHODObOOr 

Thus far we have^provid od a gl Impse Irito the under lying principles of 
. our aoaching ^sys te^ as w^ell as a simplified desGriptlon of how a 
dlrferential dipgnogtlc model caa b€ inferred from a student's behavior, 
It sboul\i now bo olear hov^ important the diagnostic model Is. to the 
^ &ucce^srnjl execution of the top^-level Issues and Examples coaQhing 
paradi.gffi , ' Ccn^equerntljrj wo feel It is important to examlhe some of the 
1 Imita^tlons , and underlying p roDlems .qT this scheme that have not yet been 
di3JC,u4aed. 'We wllL aegin with a more formal examination of Che modeling 

The ir^pu ts tb the Modeler are the a tudent* a =mo\^e and the set of better 
raovcs that the studer:it could have made. Each of these moves has assoeiated 
witH tt a set of reqvjlaite "Issues," which must be employed (in some 
.minrer) to , ob tat n that toov «, For exampla. If tKe move H was to go back 2 
spaces tc i.m4 on a shortcut, the Issues of 'SHORTCUT, SUByRACTION and 



BACKWARD are all requlredi From fch« student move, the Modeler aari Infer 
thit the student knows the lasues needed for thatmove*( 17) 

' Whab oan b© gleaned f rom nlcnowlrig ths set of fciBtter movea that the 
atudent did not . take? In general, for each better rnove M, we only know 
that at least of the set of Issues required for M was not employed and 
therefore reriecta a poten tlal weakness on the part of the gtudent. ^ But 
how do WB know whldh of these Issues blocked the atudent from making that 
raove? This is what we refer to as the '*appor tlonnjent of blame/ credit'* 
probleni: How should the Modeler apportion blaine anong the. requisite IsaJes 
for the student's failure to discover a move? ^ 

Our SDlution in WEST has been to apportion blame more or leas ^lull^ 
among ail of the Issues required for the^ miased better moves , ( 1 8 )A --sDne' 
effect of thls^ decision is the In troduc tion of tnoorreot informatiori W 
**nol3e" into the model. That is ^ blame will almost' Gertainly^ be 
apportioned to Issues that are in fact understood, . * 

Having to overcoino this source of noise is an excellent example of how 
dlagnoatng a student In a probl ^m^sol ving situation in which the atudent is 
in total Gontrdl is inherently more problernatlc than the standard 
njixed -in i ti ati ve Instructional system , In rnlxed-lnitiative systems, the 
Modeler can always construct a difrerential hypothesis fronj' this source of 
arablguity^ pose a task to the student, and see what he ^tioes, Beoause it 
aan create a sequence of suGh tasks^ each one eliminating contending 
hypotheses, the Modeler can converge on the actual afTlioting weaknesaes. 
However, such Intrusions by the Modeler into the gaming or problem-solving 
matrijc could destroy the concentration and goal directedness of the student 
Great.lng an antidote potentially more destructive than the raison-d 'etre 
for a student model .in the first place. 

The sitBplified view of a student's itiove as a sat of issues that 
soniehow \ unde rli 03 the generation of the move suggests several other areas 
of concern In the modeling process. Since the syatea does not have a 

TTTl Even this cannot be inferred if there is more than one way to derive 
the cBove and ; the "Issues^^ deal With derivational rules,. In WEST , the 
Issues are all things that uniquely underlie or are ffianifest in a roov e * 
(18) In case the Modeler has more than one. move that is better than the one 
the student made, it would be possible to find the intersection of the 
Issues required for each move. Unfortunately, the student is, in general, 
veak on naore than one Issue, so this intersection will often be emptyi 
Dieaning that at leaat two of the better moves v^ere blocked for independent 
reasons* Since the evaluators have to work with noise in any Gase,^,we did 
not inelude this noise reduGtion heuristic, It has not proven bo be a 
difriculty^ The Coach does use this strategy when ^selecting an Issue, If 
one Issue is needed for all better moves ^ it is selected" as the one most 
likely to have been ralased., • 



QOropleti * gla|a-bpx Expert, (does not aooount for th-e entire prooaga that a 
person would ua© bo derive the move ) the set of lasues does not nadeasarily 
aooount for everything required to derive the move. This optens up^ the 
posaibtllty that |h@ unberlylng reaaon the student didn't niaka a move may 
not be one of the known IssueJ at all,, but might Instead be acme other, 
sklll-that haa not been artloalaVed ag an lasue, ( 19) Any inoomplttsrtsss in 
the set of Issues^ results In more noise in the ddffirential student laodal* 
An additional aource of noise in the raodel i^s that students are seldom 
completely consistent. They often forget to use . techniques that they know 
or* get tired and aooept a move that la easy to generate, 

' Anothsr sourae of nolae Is learning. As the student plays the gaoie,^* 
we hope he will be acqulririg new skills that previously Would have shown up 
as weaknesses. Even after a student learns an Issue, his raodel will 
continue to show the weakness that hna accumulated over time* Ideally, the 
old pieces of the model should decay with tinie, Unfortunately ^ the odsts 
l^nvolvad in this Q^mputation a>e prohibitive; To avoid fihis falling of the 
raodel, the WEST Coach removes from corisiderat ion any Isaues that the 
student has used recently ( in the last three inoyes)i* 

To ^Gombat the noise which arises in the modern thte Evaluator for eaoh 
Issue is implemented as a separate procedu'rei This allows individual 
tuning of the Evaluators in response to perceived failings* In WESTf the 
Evaluators use a oornpartson of the ".taken fields** of the model with the 
"niiised fields." The comparison perdentages are adjusted to be high enough 
to yield conservative Evaluators.i This a'llevlates the problems that might 
be caused by noise for less conservative techniques. Some coaching 
opportunities may be missed but eventually if the student has a problein 
addressed by an Issue a pattern wlir eraerge* 

Strategies vs^, -Is^yj^^ 

In the scheme disbussed above> the Expert is used to create a list of 
better moves, and then the Mode ler diagnoses the student*s weakness on the 
aasurnption that he did not make any of these better raoves because he had 
not' roastered one of the requisite skills or Issues underlying them. But 
what happens if the' student is. employing a strategy different from the 
Expert's? In such cases, the reason at.udent did not make a particular 

ny J If the Coach does not have an Issue, it will not brealc in, because the' 
student's weakness may be beyond its scope. For this reason, the , Issues 
define the apace of 4y,eaknesaes the CoaQh will try to correct. 
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better movi night simply be that, he did not want to make it. Aocordlng to . 
his stpattgyp his move was the best one possible, 

In order cope With this problem, the Modeler raust be able to deteqt ' 
whtfl the student la using sorae other atrategy and to oharactefl?i preolsaly 
what this other strategy is. If an exeoutable desoription of the 
altirriatiVi strategy oari be f orned , then the Expert oari be naodlf led to use 
.the niiw atrategy. The Modeler can then reocnstruct the differential 
.. atudent model on . the basis of the nodlfied Expert In order to separate out 
what Issuea (as opposed to stratigies) the •atudent is weak on^ Eaoh of 
theae tasks has its own ajpmplicationi . Let us proeeed In thj^a dlacua^loh 
under the simplifying aasusiption that the student naintains a aonsistent 
strategy and a oonsistent set of weaknesses during the period over whloh 
the model is being created, (20) 

Dia^noaing the existence of a possi ble al ^ternatlve at rategv 

If a raodiling soheDi Looks at only one move of a student, it la 
Impossible for tt to det.erffline whether the student's failure to make 
another move steniiasd from a lack of a given ikill or from harbpring a 
suboptlmal strategy • (21 ) Hgweverp from a seauenoe of atudent moves it pay 
be poaslble to malca suoh a separation* This resulta from the fact that the 
student* a strateg^r remains the sane over the sequenGe of novfSi whereas the 
■Issues are likely to change from one move to the next* ' 

The technique for deteotlng when a student is 'using a strategy 
different from^' the Expert's involves the amount of. "te^r" in >the atudent 
model. Briefly I tear is a neaaura of- the consistenay of uae of iasues. 
Tear starts to develop when, several issues begin to reflect both a s 
substantial amount of use when, they should not have' been used and noH'^use 
when they should have been. If tear in a model gets large enough^ the 
Modeler is willing to ekperid some effort In eonjeoturlng alternative 
strategies that the 'student might be using. Any alternative strategies can 
then be tested by rerunning the Modeler over the atudent* s past noves and 
Qonparing his behavior to that ^of the Expert uain| the oonjeotured 
strategy,. If the reaulting model has subatantially less tear^i then ttte 

^ — ■ , - — ^ ^ ^ f - 

(20) A typidal period la asualiy one . aession of play, . Gonsisting of a . 
ooupls of games. Longer periods require a partitioning or layering of the 
model to oapture the change or growth of a student's knowledge. 

(21) Here again , we continue %iith th&s assumption that the Modeler is a 
watdher and not a manipulator of the environment ' and hence cannot fnterrupt 
the aotivity and post its own task, - ^ 
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oonjeoti^red sfcrattgy is taken to bp a mope aod'Urate ■appfOKanition of the 
atudeint's strategy and is^ uaed to f^rnQ the dtffe>entiWl ^lodel. If the 
reau^tlrig model isrftt substantially more cortiistent, thtn this alternative 
atra|tigy is rejected and othtr QOnjeotures ere tried fintLl all riasonable 
aonjeoturas are tested, Of course p for this cLiSsieal "g^erate and test'* 
heurtstlQ to work, not only raust the Modeler be able to gajberatt rtasonable 
alternative strategies, but the Expert also must be abre to simulate the 
strkteglea (the oonjeoturia must be runnabls by the Exper^) in order to be 
ablla to 'reoonstruot and test the resultirig student nodal 

CDiij^oturlpg plt8rnat4Ye\strat^^gJ.ea 



jadturing alternative atrategiea is extremely d/iffioult unless one 
ha/a a suffioiintly Glased world th'at the set of possible strategies oari be 
dnaraGteri^ed , Tl:ils ahiiraoteriEatlort . can take the' forro of either a 
generative meohanisra (e,g»s a grammar) that synthesizes the alternative 
strategies (Miller & Goldstein, 1 977, and also see MiHeVu article in this 
issue), or an explicit enumeration of possible alternative atrategiea. The 
world of WEST is sufficiently olosed and small enough that the latter 
techalque appears to v/ork , 

west's alternative strategies fall Into tro oategories those that 

are suboptimal because of a "mind bug" about the atruoture of the game and 
those that reflect an alteration in the sptr^it or rules of th# game. An 
example of a "mind bug" would oocur whan a student always tries to move as 
Tar ahead as is posaible given the particular spinner values a nearly 

optimal strategy but one that overlooks the potential value of bumping your 
opponent* An example of ^an alteration of the spirit of the gam.e ooGurs 
when the student is obsessed v^ith buraping his opponent (e.g., beoause of 
the pretty graphiGS effect) and will always bu*^ ^ whenever a chance arises, 
Another example that refleGts the subtle^ty of ^thts. oategory is the student 
v^ho beaomes fixated- on getting the Coadh to "spealc" or inteTaot with him, 
rhls student no longer Garea about winning the game but instead beoomes 
Involved in psyGhing ^out the actual teaching strategies embedded in the 
systen an extremely interesting "metagame", It should ,be remembered 
that the Goach is very conservative and will not break into the student* s 
gaiBe unless there la a consistent pattern of poor behavior that the Cdach^ 
pan address* If the student is' doing something corapletely "off the wall" 
it is unlikely that the Coach will break in. 
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: . Ono© a granmar or art explidlt list of. alternative strateglts is 
qre^ted, one fflay dtttrmini the 3%t of alternative strategies that a .player 
may be uiing by oriating a "handle" or ftaturirecdgnizer (similar to an 
issut recoghizer) for each strategy (or grammar rtiLi).(22) Then, as thii 
Modeler is aooruing evidend^ for perceived atudent weaknesses on Isauea^, it 
oan also be aooruing evidence on possible alternative strategies by seting 
which itrategy featurta are present in each move, These features a6t 
solely as a heuristic. They are seldom unique to a gtven strategy, as , 
levaral alternative strategies are likely to eon^ais tent with any one 
move. For exaDaple, the strategy of making a maximal nunber might prodyoe 
the same move as the strategy of maximizing t^e disbanae ahead of your 
opponent . . 

In aUmmarys these strategy features provide evtdeno^ about what 

alternative strategies the student may be liping- A strategy for which 
^ there is local evidenee. i/ then used by the Modeler to construct a new 
hypothetioal differential model*" This new model provi dea- a global' check on 
t^e.^ strategy by determining how rauoh the tear of the differehtial model has 
been :^educed ♦ " / * 

' Inv order to test the . diagnostic senaltivity of this technique to 
dist ingu3i,sh actual student weaknesses from alternative student strategies, 
we ' have Constructed various automated atudenta (an idea proposed in 
Goldstein, J977) that play' with^ specif ic weakneasea aoi simultaneously 
alternative rtrategiea. Theae testa indicate that the technique Jus)^ 
described is effective for WEST , We fully recogniie the limited nature of 
this problem for the WEST "world" and are cautious in cur belief that these, 
techniques will suffice for more bomplex; worlds. 
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M. EXPERrENCES WITH WEST 

The basie coaching system was completed in^Spring^of V975 (Burton 4 
Brown, 1 976 ). At that time, we ran an informal exper Irnent with 18 student 
teachars'j In which each one used the system for at .least one houp* 
Afterward, each was .asked to complete a questiorinalri about the^CQaoh's 
performance. -AIT but one had received advice from the Coaah. Nine of the 
teachers commented favorably about the Coaches advise. Two dthora 

XliT 5 u eh f e a £u r e recognliers can be quite complfe)c and orten^ require 
properties of the space of possible moves instead of Just the given student 
move. For ^example * one feature might qoneern whether the raove involved the 
maximuffl possible number given the particular spinners. 
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disagreed; one said that tht CoaGh was offering a gtratigy ^that he did not 
feel he should f©ll,oi# beoause it would leave him "vulntrable\ to attaek," an 
element of . strategy not known to pur bUrrent Expert , Eight d.f ten aubjeots 
found the comments helpful in learning a better way to pity t^e game and, 
most important nine of ten felt that the ^paqh* manif^atad a ^ood 
undaratandlng of ^helr yefLknesp^a ! One subject oomnented, "I ffilsundera too^d 
a rule; the computer pioked It up in the seoond game*'* \ ^ 

WEST has also been used In elementary aohool classrooms. In a 
controlled experiment ,^ the aoached version of WEST was oompare^ to an 
.uncoaohed virsion. Table 1 gives the distribution of move patterns^ for the 
ooaohed and unGoaGhed . groups . The coached students showed a oonsiderably 
greater variety of patterns, indicating that they-had aequln4d many of^ the 
more , subtle Jf^atterns and had not f^alle,n permanently into "ruts"\ that 
prevented them from seeing the relatively rare oooasiona wra^en such^/m^vas 
,were iffiportant. Probably the most surprising result from this experiment 
was that the students In 'the coached group enj.oyed playing the ga\pe 
□ohsiderably more than the unooached group * This* finding was especialiy 
significant, becauaa one of our greatest fears had been that our ooaching\ 
principles were suf f lal^ritly Ill-developed that either the Coaoh would 
interpret too often, destroying the inherent erijoyment of the game or too 
seld^om, failing to get students out of* ruts* We have not yet had the 
opportunity to explore whyj in fict^ students seem to prefer the game with 
the Coach. One interesting hypothesis is that^tlie itudenta using the 
ooaahlng version were actually engaged in a ffletagame of "psyching but" the 
Coach to get It to speak. If this rather romantia hypothesis turns out to 
be valid, it would 'open a new^ arena for conveying %Qme of the very 
important survival principles for formal education* ■ . 



PATTERN 



Table L. 

Comparison between coached and uncbachfiid 
groups o£ the percentege of timei each 
move pattern was used when it was the 
: best, move. • ' ■ ■ 

Coached Group (%) * 



( gont ro 1 Group , ( I) * 



(AtB)-C 

(A*B)+C 

(A*B)-C 

(A+B)*C 

A-(B+C) 

A*(B-C) 

(A*B),/C 

A/(B-C) 

A-(B/C) 

(A/B)-C 

.<A-B)/C 

A-(B*C) 

(A+B) It 

A/(B*C) 

A/(B+C) 

A+(B/C) 



72 




74 


S7 '., 




58 


41 




46 


65 




44 


13 ' ' 




29 


32 


• 


22 


23 . 




9 


25 , 




0 


14 




0 


14 




0 


14 




0 


13 




0 


0 




0 


0 




0 


0 




' 0' 


0 




0 



SPECIAL mm% 



.jControl Group 
TOmx 72 
BUMP I , 18 
SHORTCUT: 41 



Joached Group (%) ^ 



TOWN I 
BUtffi 
SHORTCUT ! 



79 
54 
54 



* % of time pattern was taken arid was best. 
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